Independently Extensible Solutions to the Expression Problem

The expression problem is fundamental for the development of extensible software. Many (partial) solutions to this important problem have been proposed in the past. None of these approaches solves the problem of using different, independent extensions jointly. This paper proposes solutions to the expression problem that make it possible to combine independent extensions in a flexible, modular, and type-safe way. The solutions, formulated in the programming language Scala, are affected with only a small implementation overhead and are easy to implement by hand. 1 The Expression Problem Since software evolves over time, it is essential for software systems to be extensible. But the development of extensible software poses many design and implementation problems, especially, if extensions cannot be anticipated. The expression problem is probably the most fundamental one among these problems. It arises when recursively defined datatypes and operations on these types have to be extended simultaneously. The term expression problem was originally coined by Phil Wadler in a post on the Java-Genericity mailing list [25], in which he also proposed a solution written in an extended version of Generic Java [3]. Only later it appeared that Wadler’s solution could not be typed. For this paper, we paraphrase the problem in the following way: Suppose we have a datatype which is defined by a set of cases and we have processors which operate on this datatype. There are primarily two directions along which we can extend such a system: • The extension of the datatype with new data variants, • The addition of new processors. We require that processors handle only a finite number of data variants and thus do not provide defaults which could handle arbitrary cases of future extensions. The challenge is now to find an implementation technique which satisfies the following list of requirements: • Extensibility in both dimensions: It should be possible to add new data variants and adapt existing operations accordingly. Furthermore, it should be possible to introduce new processors. • Strong static type safety: It should be impossible to apply a processor to a data variant which it cannot handle. • No modification or duplication: Existing code should neither be modified nor duplicated. • Separate compilation: Compiling datatype extensions or adding new processors should not encompass re-type-checking the original datatype or existing processors. We add to this list the following criterion: • Independent extensibility: It should be possible to combine independently developed extensions so that they can be used jointly [21]. Implementation techniques which meet the last criterion allow systems to be extended in a non-linear fashion. Such techniques typically allow programmers to consolidate independent extensions in a single compound extension as illustrated by Figure 1. By contrast, without support for independent extensibility, parallel extensions diverge, even if they are completely orthogonal [7]. This makes a joint use of different extensions in a single system impossible. This paper presents two families of new solutions to the expression problem. One family is based on objectoriented decomposition while the other is based on functional decomposition using the visitor pattern. In its original form, each of these decomposition techniques allows extensibility only in one direction (data or operations), yet disallows extensibility in the other. The solutions presented here achieve independent extensibility of data and operation extensions. They are sufficiently simple and concise to be immediately usable by programmers.